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CHE  MANY  METHODS  TO  MEASURE  TESTABILITY:  A  HORROR  STORY 


Jim  Bussert,  Test  Engineer,  Naval  Ocean  Systems  Center  Code  721 
271  Catalina  Blvd,  San  Diego,  CA  (619)  225-7667 


The  author  has  personally  modeled  designs  varying  from  a  Small 
Scale  Integration  (SSI)  printed  circuit  board  (PCB)  up  to  a 
modern  tank  Fire  Control  System  (FCS)  of  over  20  boxes  using  over 
a  dozen  Testability  Figure  of  Merit  (TFOM)  tools.  These  various 
algorithms  used  a  total  of  70  different  outputs  that  were 
intended  to  measure  the  TFOM  inherent  for  a  particular  design. 
This  paper  will  first  review  and  categorize  all  of  these  TFOM 
values.  Next,  will  be  an  attempt  to  lay  the  groundwork  necessary 
to  figure  out  if  the  optimum  TFOM  should  be  *one  magic  number"  or 
several  unique  numbers.  And  finally,  if  several,  what  could  they 
be  ? 


TFOM  TOOLS 


The  variety  of  TFOM  tools  provide  testability  output  measures 
differing  from  one  number,  up  to  a  maximum  of  24  measures.  Table 
1  summarizes  the  tools  used  by  the  author,  with  the  number  of 
measures  listed  for  each.  The  three  Sandia  Controllability 
Observability  Analysis  Program  (SCOAP)  type  TFOM  tool  outputs  are 
broken  into  controllability  (CO)  and  observability  (OBS)  of  "one" 
and  "zero"  for  sequential  and  combinational  circuits,  which  is 
five  or  six  different  TFOM  values.  When  this  breakdown  of  CO  and 
OBS  is  followed,  it  drives  the  total  TFOM  output  number  up  to  84. 
Some  TFOM  tool  outputs  include  other  information  such  as  fault 
isolation  or  dependency  flow  charts  which  are  very  useful,  but 
not  suitable  as  candidates  for  TFOM  measure  values.  It  is  not 
the  intent  to  compare  the  13  TFOM  tools  to  each  other,  but  only 
to  review  all  TFOM  outputs  as  a  database.  Therefore  there  will 
be  no  correlation  of  the  TFOM  tool  output  measures  to  the  parent 
TFOM  tool(s)  that  they  came  from,  although  some  are  unique  to  one 
TFOM  tool.  Table  1  only  indicates  the  tip  of  the  TFOM  measure 
forest. 


CHE  TFOM  FOREST 


The  first  step  is  to  list  each  of  the  testability  measures  from 
every  tool.  This  will  result  in  a  summation  of  possible  test 
measures  that  need  to  be  reviewed  to  determine  the  optimum 
candidates  for  the  best  of  the  group.  To  avoid  needless 
repetition  and  make  make  an  understandable  and  manageable  table, 
two  liberties  were  taken.  Seven  items  that  appear  more  than  once 
shall  be  listed  only  one  time  at  the  top,  and  CO  and  OBS  shall  be 
listed  without  all  of  the  sequential  and  combinational  "one"  and 
"zero"  variations.  The  resulting  49  measures  are  shown  in  Table 
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2.  First,  70  of  the  output  measures  have  been  listed  from  all  13 
TFOM  tools.  Duplicate  or  redundant  figures  are  noted  in  order  to 
reduce  the  total  number  in  Table  2  down  to  49  unique  TFOMs . 


The  seven  values  that  are  used  by  more  than  one  TFOM  tool  are 
listed  in  Table  2,  along  with  the  number  of  times  that  each 
appeared.  Logically,  it  would  seem  that  the  more  often  that  a 
particular  output  was  used  by  various  algorithms,  the  stronger 
its  chances  of  being  a  valuable  TFOM  candidate.  This  may  not 
necessarily  be  true.  Ambiguity  group  size  leads  with  seven 
occurrances.  Next  in  number  is  recommended  test  point  locations, 
with  five  times.  Next  in  useage  are  feedback  loop  report  and 
list  of  components  with  four  each.  Three  measures  are  tied  with 
three  occurrances  each.  These  are  observability, 

controllability,  and  inherent  testability  figure  of  merit,  or  the 
"one  value  FOM" .  Unnecessary  test  point  locations  are  an  output 
two  times.  Finally,  there  are  48  other  TFOM  output  results  that 
appear  only  one  time. 

Next,  the  remaining  49  total  unique  outputs  will  be  subjectively 
categorized  as  to  whether  they  are  potential  strong  "FOM  number" 
candidates.  These  will  be  prime  candidates  under  consideration 
as  the  "one  magic  number".  These  would  be  of  most  value  to  a 
buyer,  such  as  the  government,  who  demands  a  simplistic  and 
easily  specified  desired  level  of  testability.  This  selection  is 
admittedly  subjective,  but  the  first  cut  of  candidates  for  one 
FOM  are  shown  in  column  two  of  Table  2.  All  occurrances  of 
observability,  controllability,  and  FOM,  as  well  as  three  of  the 
seven  ambiguity  group  types  were  selected.  Additionally,  seven 
other  one-time  outputs  are  potentially  good  FOMs.  This  results 
in  a  total  of  11  attractive  single  TFOM  nominees. 

The  third  column  of  the  table  shows  output  lists,  rather  than 
single  numbers  or  values.  These  lists  are  of  great  value  to  the 
designer,  but  being  lists,  cannot  be  considered  for  "one  number" 
FOM.  Frequently  these  lists  clarify  or  detail  related  FOM 
numbers,  such  as  a  list  of  items  in  an  ambiguity  group  or  list  of 
BIT  candidate  sites.  In  fact  those  two  lists  accounted  for  nine 
occurrances  in  the  first  duplication  elimination  task. 
Additionally  four  of  the  seven  ambiguity  group  examples  were 
lists. 

The  last  column  in  Table  2  is  all  of  the  other  remaining  TFOM  — 
values.  These  may  be  lists  or  single  numbers  that  have  some  ® 
value  regarding  the  design,  but  appear  to  be  of  less  importance  O 
or  less  critical  than  the  TFOMs  previously  noted  in  the  second  O 
and  third  columns.  This  final  step  is  hard  to  do,  and  it  would  — 
be  much  easier  to  leave  a  majority  of  the  TFOM  outputs  in  the  ~~ 
other  two  columns.  However,  such  an  approach  would  not  make  the  — . 
needed  progress  toward  the  important  goal  of  reducing  the  present 
TFOM  forest.  The  current  49  TFOMs  that  a  user  is  faced  with  is  1B 
overwhelming  and  confusing.  These  values  will  continue  to  — 
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multiply  as  more  TFOM  and  Design  For  Test  (DFT)  tools  become 
available.  System  buyers  who  must  have  satisfactory  and 
meaningful  specifications  of  testability  as  targets  for  designs, 
need  a  succinct,  useful  and  optimum  TFOM  criteria. 

ONE  MAGIC  NUMBER  OR  SEVERAL  NUMBERS 

Finally,  a  close  scrutiny  of  the  strong  TFOM  candidates  is 
required.  This  gets  into  an  area  potentially  as  volatile  as 
politics  or  religion.  That  is  the  pros  and  cons  of  one  number 
versus  several  numbers  as  a  design  TFOM.  This  is  followed 
immediately  by  the  decision  as  to  which  one  number  is  the  magic 
testability  holy  grail.  This  issue  is  potentially  so 
controversial  ,  that  the  author  purposefully  avoided  it  while 
doing  dozens  of  TFOM  evaluations  on  three  benchmark  designs.  It 
would  not  serve  the  cause  of  testability  to  get  into  a  dog  fight 
over  the  "one  acceptable  FOM"  and  stall  utilizing  all  of  the 
assets  available.  Curiosity  drove  to  the  making  of  a  list  of  all 
TFOM  outputs  personally  used  over  the  last  few  years.  This  list 
of  69  seems  to  force  test  professionals  to  finally  face  this 
issue  head  on  in  a  positive  dialogue  and  attitude.  Testability 
obviously  has  many  features  and  effects  that  can  be  noted  and 
measured  or  listed,  so  it  seems  overly  simplistic  to  assign  only 
one  "magic  number"  as  a  viable  design  goal.  Different  design 
technologies  such  as  digital,  analog,  machanical  or  hybrid  do 
have  their  own  unique  TFOM  differences  which  one  number  could  not 
possibly  wrap  its  arms  around.  Possibilities  to  pursue  could 
include  different  TFOM  criteria  for  each  technology,  or  the 
summing  of  several  important  TFOM  values  into  one  number. 

PICKING  THE  BEST  TFOM  TREES 

Referring  back  to  the  initial  high  repeater  items  as  potential 
magic  FOM  nominees,  it  turned  out  that  five  of  the  eight 
ambiguity  group  items  are  lists,  as  were  the  next  two  most 
numerous  items,  test  points  and  feedback  loops.  So  it  turns  out 
that  the  three  big  repeaters  are  not  possible  FOM  type  values. 
Selection  of  a  single  value  such  as  controllability, 
observability,  or  ambiguity  group  size  would  overlook  other 
critical  DFT  problem  areas.  A  problem  with  currently  available 
single  FOM  outputs  are  questions  on  their  objectivity,  and  what 
do  they  really  say  or  prove  ?  One  solution  to  the  quandary  of 
several  good  measures  that  cover  the  various  and  differing  areas 
of  DFT  concern,  is  to  sum  several  into  one  mean  or  average  value. 
That  sum  value  could  be  a  TFOM  goal  of  known  meaning  and 
discipline. 

TEST  PROGRAM  SET/SYSTEM  MEASURES 


Long  respected  TFOMs  of  a  sort  have  been  the  Test  Program  Set 
requirements  and  system  design  specs  for  percent  of  coverage  and 
percent  of  ambiguity  group  sizes  allowed.  MIL-STD-2077A 
specifies  general  requirements  for  test  program  sets.  One  good 
measure  in  2077  is  Test  Program  Comprehension  (TPC) ,  which  is  the 


ratio  between  the  number  of  faults  detected  and  the  total  number 
of  faults.  The  MIL-STD  states  TPC  will  be  a  minimum  fault 
detection  of  95  percent.  Another  MIL-STD  TPS  measure  is  time  to 
isolate  (TTI)  for  the  worst  case  logic  chain  of  each  diagnostic 
branch.  Although  TPC  or  maximum  logic  chain  TTI  are  not  listed 
among  the  69  TFOM  tool  outputs,  analagous  values  appear  once  for 
each.  System  fault  isolation  requirements  for  specified 
ambiguity  group  size  to  a  percent  of  faults  are  a  common  system 
design  parameter.  MIL-STD-470  Maintenance  Demonstration  (M-Demo) 
analyzes  for  design  fault  isolation  capability,  but  no  parameters 
are  given.  The  ambiguity  group  isolation  percent  criteria  are 
not  stated  in  any  MIL-STD,  but  are  common  design  specifications. 

It  is  surprising  that  these  TPS  or  system  specification  type 
values  were  very  rare  in  the  69  TFOM  values.  Percent  of  signal 
coverage,  Mean  Time  to  Isolate  (MTTI) ,  and  non-detect  list,  which 
are  common  TPS  or  system  specs ,  each  appeared  only  one  in  the 
total  of  69  items.  In  fact,  the  MTTI  could  conceivably  have  been 
included  in  the  recurring  group  under  two  or  three  occurrances. 
One  tool  had  MTTI  stand-alone,  another  coupled  MTTI  with  MTTR, 
and  a  third  tool  had  average  number  of  tests  to  ilolate,  which 
could  be  converted  to  MTTI  in  minutes.  To  keep  a  standard 
methodology  for  exact  match-up  of  values  before  considering 
duplication,  these  three  outputs  were  not  interpreted  as  being 
the  same.  Objective  process  is  a  difficult  discipline  in  such  a 
subjective  and  volatile  area. 

OPTIMUM__TFOM  MEASURE  CANDIDATES 

Although  it  is  not  the  intent  of  the  author  to  add  more  FOMs  to 
the  initial  list  of  69,  it  would  seem  that  the  time-proven  values 
used  in  TPS  or  system  design  specs  are  good  TFOM  candidates. 
Prime  nominees  for  quality  TFOM  factors  are  illustrated  in  Table 
3,  with  TFOM,  TPS,  and  system  measures  listed  for  correlation. 
Note  that  only  ambiguity  group  size  and  percent  of  coverage  are 
used  in  all  three  TFOM  measure  source  columns.  The  summing  or 
averaging  of  several  of  the  better  TFOMs  noted  here,  would  be  a 
good  basis  for  a  meaningful  and  useful  FOM  design  goal.  It  would 
also  help  link  the  different  test  design  disciplines  into  some 
common  terms  and  measures,  which  is  a  good  goal  all  by  itself. 

CONCLUSIONS 

TFOM  tool  vendors  can  no  longer  continue  to  make  up  their  own 
unique  TFOMs  with  no  regard  for  the  need  of  standard  design 
benchmarks,  which  the  designer  and  design  buyer  need  in  their 
designs.  Other  contractual  "ilities”  such  as  reliability  or 
maintainability  have  standard  constant  measures  that  everyone 
agrees  upon  and  comply  with.  Until  testability  does  the  same 
thing,  it  cannot  be  designed  or  measured  in  an  objective  and 
disciplined  manner.  In  most  cases,  the  design,  TPS,  and  system 
spec  are  all  measuring  the  same  thing  by  different  criteria.  For 
example,  a  Computer  Aided  Design  (CAD)  workstation  with  SCOAP 
software  will  seek  observability  and  controllability,  then  the 
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TPS  has  different  MIL-STD  test  coverage  specs,  and  the  final 
system  has  still  different  MTTI  and  availability  goals  to  meet. 

The  proper  use  of  any  of  the  available  TFOM  tools  by  a  designer 
will  undoubtedly  improve  the  testability  of  a  design  and  reduce 
test  program  costs  and  fault  isolation  time.  It  is  also 
probable,  that  until  this  TFOM  question  is  faced  head  on,  TFOM 
tools  will  continue  to  crank  out  additional  dozens  of  new  and 
different  unique  numbers  and  lists.  Most  tragically  of  all, 
designs  of  doubtful  testability  will  continue  to  be  bought  off  by 
the  government,  company,  or  contractor  until  meaningful  and 
agreed  upon  TFOM  measures  are  adopted  and  used  by  designers. 


TOOL 


NUMBER  OF  OUTPUTS 


MIL-STD-2165 

ONE 

RADC  LIST 

ONE 

ITFOM 

ONE 

LONGENDORFER 

FOUR 

LOGMOD 

FIVE 

CALMA  COPTR 

FIVE 

SCOAP 

SIX 

DAISY  DTA 

SIX 

ASTEP 

SIX 

ACE 

EIGHT 

CAFIT 

EIGHT 

IN-ATE 

NINE 

STAMP 

TWENTY 

FOUR 

Table  1.  Overview 

of  13  TFOM 

tools  with 

the  number  of  test 

measure  outputs  of  each. 

i 

IS4 

yA 


MEASURE 


USED  IN  TFOM  USED  IN  TPS  DESIGN  SPEC 


CONTROLLABILITY 

YES 

NO 

NO 

OBSERVABILITY 

YES 

NO 

NO 

AMBIGUITY  GROUP 

YES 

YES 

YES 

ONE  NUMBER  FOM 

YES 

NO 

YES 

PERCENT  OF  COVERAGE 

YES 

YES 

YES 

MEAN  TTI  (MTTI) 

YES 

NO 

YES 

NON-DETECTS 

YES 

YES 

NO 

MAXIMUM  TTI 

NO 

YES 

NO 

T.P.  COMPREHENSION 

NO 

YES 

NO 

Table  3.  Optimum  TFOM  measures  which  come  from  TFOM  tools,  TPS 
acceptance  specs,  or  system  design  specs. 

TFOM  OUTPUT 

USED 

BEST  FOM 

BEST  LIST 

OTHER 

AMBIGUITY  GROUP  SIZE 

8 

3 

5 

TEST  POINT  SITE  HEIRARCHY 

5 

5 

FEEDBACK  LOOPS 

4 

4 

OBSERVABILITY 

3 

3 

CONTROLLABILITY 

3 

3 

INHERENT  FOM 

3 

3 

UNUSED  TEST  POINTS 

2 

2 

ITEM  INVOLVEMENT  RATIO 

1 

1 

MTTI/MTTR 

1 

1 

MEAN  REPLACEMENT  LIST  SIZE 

1 

1 

MEAN  PRIORITIZED  REPLACE  POSITION 

1 

1 

PROBABILITY  OF  FAULT  DETECTION 

1 

1 

MTTI 

1 

1 

DETECTED  FAILURE  RATE 

1 

1 

BASIC  TEST  PROGRAM 

1 

1 

ATLAS  TEST  PROGRAM 

1 

1 

EDIF  FILE 

1 

1 

TEST  STRATEGY  FLOWCHART 

1 

1 

RTOK  FREQUENCY 

1 

1 

DIAGNOSIS  AVERAGE  COST 

1 

1 

AVERAGE  REPLACEMENT  COST 

1 

1 

ISOLATION  LEVEL 

1 

1 

TEST  LEVERAGE  (TL) 

1 

1 

NON-REDUNDANT 

1 

1 

TEST  UNIQUENESS 

1 

1 

TEST  FEEDBACK  DOMINANCE 

1 

1 

COMPONENT  FEEDBACK  DOMINANCE 

1 

1 

NONDETECT  PERCENT 

1 

1 

HIDDEN  FAILURE  MEASURE  (HFM) 

1 

1 

INPUT  MODIFIED  HFM 

1 

1 

PERCENT  HFM  (PHFM) 

1 

1 

INPUT  MODIFIED  PHFM 

1 

1 

FALSE  FAILURE  MEASURE  (FFM) 

1 

1 

INPUT-MODIFIED  FFM 

1 

1 

DEPENDENCY 

1 

1 

TEST  INTERDEPENDENCY 

1 

1 

TEST  DEPENDENCY 

1 

1 

FALSE  ALARM  TOLERANCE 

1 

1 

THEORETICAL  MIN/MAX  TL 

1 

1 

EXTERNAL  DEPENDENCY 

1 

1 

EXCESS  TEST  MEASURE 

1 

1 

NEGATIVE  RECONVERGENCE 

1 

1 

LOGIC  REDUNDANCY 

1 

1 

DEVICE  SIGNAL  COVERAGE 

1 

1 

TPS  FLOWCHART 

1 

1 

AVERAGE  NUMBER  TTI 

1 

1 

TEST  SIGNAL  COST  CONTRIBUTION 

1 

1 

TEST  POINT  COST  CONTRIBUTION 

1 

1 

BLOCK  TFOM 

1 

1 

Table  2.  Summary  of  70  TFOM  outputs 

from 

13  tools, 

with  values 

and  lists  categorized  and  prioritized. 
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